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Executive Summary 



Chile has introduced a system of merit awards to schools called the National System to Evaluate 
School Performance (Sistema Nacional de Evaluacidn del Desempeflo de los Establecimientos 
Educacionales Subvencionados or SNEO). The SNEO system has been carefully designed and 
well implemented. It avoids many of the problems associated with merit pay for individual teach- 
ers. As it enters its third round (every two years) of measurement and awards, it appears to be ac- 
cepted by key actors in the education sector. This paper provides information on the characteristics 
of this approach to providing incentives to improve the quality of education. 

Key features of the SNED system include the following: 

Awards go to school establishments, not individual teachers. This avoids many of the prob- 
lems associated with merit pay to individual teachers. 

Schools are stratified so that competition is between relatively comparable establishments. 

Separation of schools into homogenous groups makes the competition more equitable. Teachers 
feel this is desirable. 

Awards go to schools constituting 25 percent of the enrollment in each stratum. Ninety per- 
cent of the awards are paid directly to all the teachers as bonuses. School directors may allocate 
the remaining 10 percent to outstanding teachers. 

Chile's national student assessment system (SIMCE) provides an important part of the ba- 
sis on which school performance is evaluated. SIMCE has been in operation for over a dec- 
ade and is relatively well accepted in the education community. Its existence made the cost of es- 
tablishing the SNED system reasonable. 

Awards are based on an index composed of six factors. Each factor is made up of one or 
more indicators. Absolute levels of student learning, as measured by SIMCE, is one factor and 
counts for 37 percent of the total weight in the index. Changes in SIMCE scores since the last ap- 
plication of the tests is a second factor and counts for 28 percent of total weight in the index. The 
remaining factors constituting 35 percent of the weight are also largely based on education out- 
comes. 

The awards are fully competitive. That is to say, schools may win repeatedly. 
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Chile's Ministry of Education has commissioned careful evaluations of the first two rounds 
of SNED. Changes have been made in the composition of the factors in the index and in their 
weights. 

Teachers and school directors express reasonable satisfaction with the SNED system. One 
of the aspects of the evaluation sought opinions of teachers and directors. Reactions were gener- 
ally favorable. The teachers' union does not object to the SNED system. 

The distribution of SNED awards across schools receiving voucher payments, that is to say across 
both municipal (public) schools and private-subsidized schools, is roughly equal, although a slightly 
higher percentage of private-subsidized schools won awards. Municipal schools account for 69.1 
percent of all schools receiving vouchers but received only 67.5 percent of the SNED awards, 
while private-subsidized schools represent 30.9 percent of all eligible schools and won 32.5 per- 
cent of the awards. 

A system of merit awards to schools involves lower information and transaction costs than merit 
pay to individual teachers. The incremental cost of establishing the SNED system in Chile was low 
because the country’s relatively high-quality system of standardized tests, covering 100 percent of 
the students at each level tested, was already in place and generally accepted in the education 
community. 

One problem that may arise is how to manage the awards over time. The SNED system is fully 
competitive and schools can win awards repeatedly (although it is more difficult to show improve- 
ment in test scores as a school’s average scores become higher.) There is a danger that teachers 
in schools that win awards in one application of SNED will be frustrated if their school do not win in 
subsequent rounds of SNED measurement and awards, or that teachers in schools experiencing 
great difficulties will lose hope that they can ever win and therefore not make efforts to improve. 

Another potential problem would arise if teachers altered their behavior in undesirable ways order 
to win awards, such as focusing teaching efforts only on the subjects tested in SIMCE. Although 
there have not been indications of opportunistic behavior on the part of teachers, such behavior 
could emerge as teachers become better informed about SNED. 
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Introduction 



There is widespread interest in both developed and devetoping countries in improving the effi- 
ciency and effectiveness of education systems. This interest has given rise to the idea of intro- 
ducing market-like incentives into the education sector, bringing to bear the powerful forces of 
competition and the price system in order to enhance school performance. This has led to policy 
recommendations and experimental programs to encourage teachers to improve their perform- 
ance, to enable parents to choose their children's school and thereby introduce competition into the 
marketplace for education, and to encourage development of programs that promote innovation 
and permit freedom from bureaucratic control. Introducing market incentives has met with success 
in some social sectors, especially health, and in other activities once considered public-sector re- 
sponsibilities, such as prisons. Elementary and secondary education, however, have proved rela- 
tively resistant to attempts to introduce prices and markets. 

Chile is experimenting with a "National System to Evaluate School Performance” (Sistema Na- 
tional de Evaluatidn del Desempefio de los Establecimientos Educacionales Subvencionados or 
SNED) that provides merit awards to all the teachers in a school based largely on student out- 
comes. SNED differs from systems of merit pay for individual teachers. This paper provides a de- 
scription of the SNED program and an analysis of how it is functioning. It is based on the work of 
the developers of SNED at the Chilean Ministry of Education, on evaluation studies of SNED 
commissioned by the ministry, and on a study commissioned by the Inter-American Development 
Bank. 1 

The SNED program, established on the basis of Law No. 19.410 of 1995, has carried out two 
rounds of measurement and awards since its inception (1996-97 and 1998-99) and is making 
preparations for the third round of measurement and awards for 2000-2001 . The descriptive in- 
formation about SNED in this paper relates mainly to the most recent application of SNED (1998- 
99). Information in Parts 2 and 3 on the reactions of school directors and teachers comes from the 
evaluation study conducted after the 1996-97 application of SNED. At this stage it is possible to 
comment on the design and implementation of SNED, although it is still too soon for the system to 



1 Ministerio de Educacidn (1998): Heyl, Vivian and Marcela Guzman. “Evaluadbn del Desempefio, SNED," 
Santiago, Chile: Ministry of Education (processed). Ministerio de Educadbn (1998): Reconodmiento aj 
Compromiso Docente . Ministerio de Educaddn (1997): Evaluation of SNED 1996-97, commissioned by the 
ministry, performed by Alejandra Mizala and Pilar Romaguera: “Evaluadbn de la Implementad6n y Resultados 
del SNED, 1996-97.” Centro de Economla Aplicada, Facultad de Ingeneria Industrial, Universidad de Chile 
(processed). Mizala and Romaguera (1999) "Sistemas de Incentivos en Educadbn y la Experienda del SNED 
en Chile". Trabajo produddo como parte del proyecto "Los Maestros an America Latina: Carreras e 
Incentivos" de la Red de Centres de Investigaddn del Banco Interamericano de Desarrollo. Paper presented 
at an IDB conference in San Jos6, Costa Rica, June 1999. 
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have had a demonstrable impact on student learning, either in schools that have won awards or in 
the system as a whole. 
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A System for Evaluating School Performance 



The Context of SNED 

Several factors present in the Chilean context have important influences on the SNED system and, 
in the case of Chile's testing system, on the information costs of SNED. 



Voucher System 

Chile’s education system has been the subject of extensive study, especially because of its na- 
tional voucher system. 2 The military government decentralized control of schools to the municipal 
level beginning in 1980 and permitted establishment of private schools that receive the same 
voucher payments as municipal schools. Voucher payments are based on the number of students 
in attendance (verified by monthly on-the-spot checks) and the per-student stipends are equal for 
both municipal and private-subsidized schools. (Elite private schools whose unit costs are higher 
than the level of the voucher are excluded from receiving vouchers.) In 1997 there were 1.84 mil- 
lion children enrolled in municipal schools that received voucher payments and 1.1 million students 
in the private-subsidized schools. 



Testing System 

Another feature of Chile’s education system, which is especially important for the SNED program, 
is its System for Measuring the Quality of Education (Sistema de Medicibn de la Calidad de la 
Educacibn, SIMCE.) Introduced in 1988, SIMCE is a standardized test of Spanish and mathemat- 
ics skills administered every two years to students at grades 4 (in even years) and 8 (in odd years) 
at the level of basic education. 3 More recently SIMCE tests also have been given at the second 
year of secondary education in 1994 and 1999. SIMCE provides information on student achieve- 
ment. The tests are standardized, which makes possible meaningful comparisons between aver- 



2 Recent publications on Chile’s voucher system include: Camoy, Martin and Patrick McEwan. "Choice be- 
tween Public and Private Schools in Chile". Stanford University, School of Education, November, 1998 (proc- 
essed). Taryn Rounds Parry. “Theory Meets Reality in the Education Voucher Debate: Some Evidence from 
Chile”. Education Economics, vd. 5 no. 3 (December. 1997), pp. 30-32, and Varun Gauri, School Choice in 
Chile: Two Decades of Educational Reform . University of Pittsburgh Press, Latin America Series, 1999. 

3 An earlier system of standardized tests, the Programs de Evaluadbn del Rendimiento Escolar (PER), was 
applied at the fourth and eighth grade levels from 1982 to 1984. 
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age SIMCE scores for a school between successive applications. Testing is almost universal but 
reporting is on the basis of school average scores, not individual student scores. 

Education Reform Program 

Chile has undertaken successive waves of education reform. 4 One of these, in the 1980s during 
the military regime, brought about the decentralization/privatization of education and establishment 
of the voucher system. The most recent reform has come about since the return of democracy in 
1990 and focuses on improving the content and quality of education. This current major reform 
effort has four main aspects or “pillars”: (1) extending the length of the school day, including the in- 
frastructure investment to make the longer school day possible (double shifts are eliminated, which 
creates an immediate demand for classroom places); (2) programs to improve teaching and learn- 
ing; (3) curricular reform; and (4) strengthening the teaching profession. Along with an extensive 
program of in-service teacher training, the SNED system of evaluation and merit awards is part of 
the fourth “pillar” of the reform: strengthening the teaching profession. 

World Bank Education Projects 

Since 1990 the World Bank has supported several important education projects in Chile. The first, 
known by the acronym MECE (Mejoramiento de la Equidad y Calidad de la Educacidn), financed 
improvements at the primary level and the second (informally called MECE Media) addressed sec- 
ondary education. 5 An important aspect of the MECE projects was financing for small capital 
grants to schools, called Educational Improvement Projects (Proyectos de Mejoramiento Educa- 
tivo, PMEs). PMEs were initially awarded on a competitive basis, but eventually all schools were 
able to win awards. The first application of SNED took account of the process by which a school 
prepared its PME but this has not been considered in later applications. These projects have 
stimulated educational innovations and have been part of a process that has raised the issues of 
educational equity and quality to the top of the country's policy agenda. 

Description of the SNED System 
General Design 

SNED provides merit awards to basic and secondary school establishments. The funds must be 
used for bonuses to teachers. The awards are based on a school's performance as measured by 




4 A detailed analysis of Chile's education reform experience is provided in: Frangoise Delannoy. “Education 
Reforms in Chile, 1980-1998: A Lesson in Pragmatism.” Country Studies. Education Reform and Manage- 
ment Publication Series . Vol. I, No. 1 (June 2000). 

s A third project for higher education has been approved but has little or no bearing on SNED. 
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an index comprised of six factors. The index includes the absolute levels of SIMCE scores and 
improvements since the last SIMCE tests, as well as other indicators of educational outcomes and 
a few process measures. Awards are made to the best-performing schools in each of a number of 
strata, which means that competition is between relatively comparable schools. SNED evaluations 
take place every two years and the awards are distributed over a two-year period. Any municipal 
or private-subsidized school is eligible to win an award, and may win in successive evaluations. 
That is to say, SNED awards are fully competitive. The information in the following sections is 
based on current procedures and incorporates changes implemented following two studies of the 
system that took place following the 1996-97 round of SNED. 6 

Awards Based on Index Scores 

One of the interesting features of the SNED system is the use of quantitative indicators combined 
into an index as the basis for the awards. Data come from the SIMCE tests and a questionnaire 
given to parents at the time the tests are administered, a special survey carried out for the purpose 
of SNED, a report associated with subvention or subsidy payments to the school, and the Ministry 
of Education statistics unit. The index of school excellence includes six factors in which outcomes 
weigh heavily. Table 1 below presents these factors and the indicators used in calculating each 
factor. 

The indicators used in calculating each of the factor scores have equal weight and are combined to 
obtain the school’s rating in terms of each factor. There were changes in the indicators used to 
evaluate each factor between the 1996-97 and the 1998-99 applications of SNED. The system 
continues to be subject to modifications as changes in external circumstances or in policy priorities 
occur. 



8 Information in this section is derived from the following documents: Ministerio de Educacfon, Santiago, Chile: 
Heyl, Vivian and Marcela Guzman (1998), op. at. Ministerio de Educacidn, Departamento de Comunicaddn 
Social (1999). Reconodmiento al Compromiso Docente . Ministerio de Educaddn: Mizala, Alejandra and Pilar 
Romaguera (1997), op. cit., as well as from Mizala and Romaguera (1999), op. at. (study commissioned by 
the IDB.) 
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Table 1: i 


Specifications of Factors and Indicators Used in the SNED Index, 1998-99 


Factor 


Indicators 


Effectiveness 


- Average SIMCE scores in cognitive area (Spanish & mathematics) 

- Integration of handicapped children into working life, with follow-up* 

- Mainstreaming handicapped students into regular classes, with follow-up* 


Improvement 


- Change in SIMCE scores, cognitive area, since last application 


Initiative 


- Establishment of council of teachers with meetings at least once each month 

- Monthly participation of rural teachers in micro-center meetings** 

- Development of "technical-pedagogical" activities in groups at least once each month 

- Establishment of a students’ center and meetings at least once each month 

- School has an institutional plan ("proyecto educativo") for medium- to long-term de- 
velopment 

- School has arrangements with local employers to aid handicapped students' entry into 
employment* 

- Teaching practices that include work experience for handicapped* 

- Development of teachers’ workshops 


Improvement in 
Working Condi- 
tions 


- School has full complement of teachers 

- Use of substitute teachers when regular teachers are absent 


Equality of Op- 
portunity 


- Rate of retention (staying in school) of students 

- Rate of promotion of students 

- School practices differential grouping to help special needs students 

- Does the school practice discrimination (e.g. expulsion of students who repeat, of 
students who become pregnant, or expulsion of students during the school year)?*** 

- Incorporation of students with multiple deficits* 

- Incorporation of students with severe deficits* 


Integration of 
Teachers, Parents 
and Guardians 


- Parents and guardians satisfied with ("accept") the work of the teachers 

- Establishment of a parents' center 



Source: Ministry of Education (1998): Heyl, Vivian and Marcela Guzman. H Evaluad6n del Desempefto: SNED", Cuadro 
N. 1, p. 8. Santiago, Chile: Ministry of Education (processed). 

* Indicator used only for special education schools 
** Indicator used only for rural, one-, two- or three-teacher schools 
*** Discriminatory practice enters as a negative indicator 



The factors themselves are weighted according to the scheme presented below. The weights 
given to each of the factors were subject to adjustment between the first and second rounds of 
SNED, based on the thorough-going evaluation commissioned by the ministry and carried out by 
the Centro de Economia Aplicada in 1997. Table 2 below shows the weightings in the two appli- 
cations of the SNED system. 



The main changes between the first two iterations of SNED are the important increase in the 
weight accorded to the factor “equality of opportunity” and the decrease in the total weight of 
SIMCE scores (both absolute and change since last administration) from 70 to 65 percent of the 
total value of the index. The changes were in response to the findings and recommendations 
emerging from the evaluation of SNED after the first round of application. Documents relating to 
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the SNED system emphasize that the design of the SNED system is and should be flexible and 
subject to change between applications . 7 



Table 2: Weight of Factors in Two Applications of the SNED Index (in %) 


Factor 


1996-97 


1998-99 


Effectiveness 


40 


37 


Improvement 


30 


28 


Initiative 


6 


6 


Improvement in working conditions 


2 


2 


Equality of opportunity 


12 


22 


Participation by parerrts/guardians and teachers 


10 


5 



Source: Ministerio de Educaddn, Departamento de Cornunicadbn Social (1999). Reconocimiento al Compromise) Docente . 



Stratification and Competition Within Homogenous Groups of Schools 

An important aspect of the SNED system is that it establishes competition between schools that 
are roughly comparable in terms of the student populations they serve, the socioeconomic levels of 
the communities in which they are located, and other external factors that affect student outcomes. 
For this the schools are organized into "homogenous groups" in a two-stage process. First, 
schools are divided into four main categories: urban basic and rural basic schools, and urban sec- 
ondary and rural secondary schools (the last can include schools that have both basic and secon- 
dary levels). Schools that provide only special education constitute a fifth category, but are evalu- 
ated on the basis of the index. There are sub-groups of schools within each category that are 
evaluated on the basis of different criteria: rural primary schools with single teachers or multigrade 
teaching, adult education schools, schools in penal institutions and free-standing pre-schools. The 
evaluation criteria for these subgroups are complex and differ depending on the type of school and 
level of education. For purposes of brevity and focus, this paper will not deal with them at length. 

Once grouped into categories, schools within each category are divided into “homogenous groups" 
on the basis of statistical cluster analysis. This analysis considers variables that influence the 
schools’ performance but are outside their control, such as average family expenditure on educa- 
tion, level of education of the parents, community characteristics, physical accessibility of the 
school and an “index of vulnerability” compiled by the entity responsible for distribution of scholar- 
ships and student aid (Junta Nacional de Auxilio Escolar y Becas or JUNAEB). The vulnerability 
index classifies the community the school serves according to the socioeconomic level of its fami- 
lies. 



Ministerio de Educacibn: Reconocimiento al Compromiso Docente . op. dt., p. 5; Heyl and Guzman, 
“Evaluadbn del Desempefto (SNED): Establecimientos Educadonales Subvencionados”. July 1998 
(processed), pp. 4, 11-12. 
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Schools are grouped using cluster analysis, which is described as follows: 

Cluster analysis is a statistical method that considers the socioeconomic and 
location variables corresponding to each educational establishment and, on this 
basis, groups the schools by minimizing the distance between the internal 
characteristics of each group and maximizing the distance between groups. 

The key concept in this method is that of distance, which in statistics is associ- 
ated with variance. This method, through successive approximations, mini- 
mizes intragroup variance and maximizes intergroup variance, leading to an 
optimal grouping. 8 

SNED establishes homogenous groups within each of Chile’s 1 3 regions in each of the categories 
above (urban basic, rural basic, and urban and rural secondary). Breakdown by region means that 
there is even greater similarity between schools in each stratum or comparison group than if the 
grouping process were nationwide. If there are fewer than ten schools in one of the four categories 
in a given region, there is only one group in this category. If there are as many as ten but less than 
a hundred schools, two groups are formed. If there are between 100 and 500 schools, three 
groups are formed and if there are five hundred or more schools, there are four groups in that 
category. 

Awards 

SNED ranks schools within each group according to their scores on the index and gives awards to 
schools in rank order, up to the point where the enrollment in the winning schools accounts for 25 
percent of the enrollment in that group. Award funds may be used only for bonuses for teachers 
(including head teachers or school directors). Ninety percent of the funds are distributed to teach- 
ers on the basis of the number of hours worked. (This the most equitable basis for distributing 
awards, especially for secondary schools, where teachers may work less than a full day in a par- 
ticular school. If primary school teachers in a school all work full time, the awards to each are 
equal.) The school director may allocate the remaining ten percent to the teachers who have 
made the most “outstanding” contribution to professional performance. The special awards made 
from the unallocated ten percent are explicitly designed to overcome the “free rider” problem of 
teachers who make little effort on behalf of the school’s improvement program. Some documents 
indicate that the distribution of this final ten percent is based on decisions of “the education profes- 
sionals in the establishment.” 9 Other studies say the school director makes the decision, and that 
this has given rise to some controversy. 



8 Translated from Heyl and Guzman, op at., p. 6. 

9 Ibid. p. 3. 
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Approximately 31 ,000 teachers have received bonuses in each of the first two rounds of SNED 
awards. The average amount of the bonus per teacher per year was 225,000 Chilean pesos in 
1999 (equal to approximately US$450 per year). The relatively low amount of the awards has the 
effect of keeping the SNED process from becoming too much of a “high-stakes” exercise. This re- 
duces the incentives for opportunistic behavior on the part of the teachers, such as focusing 
teaching attention on only those subjects evaluated by SIMCE. Evaluation of SNED after one 
round of competition did not indicate instances of opportunistic behavior, but the same evaluation 
found that most teachers (even in winning schools) did not know much about SNED. In future 
rounds there may be more of a tendency toward opportunism. 

Winning schools are identified every two years. SNED makes award payments once each quarter 
during the two year period. The number of establishments that won awards was 2,285 during the 
first two-year period, decreasing to 1,826 during the 1998-99 SNED exercise. Total payments 
have increased slightly each year, reflecting both an increase in enrollments and a modest adjust- 
ment for price changes. The total payment in 1998 was 6,493 million Chilean pesos (roughly 
US$16.2 million) or approximately one percent of the total amount of payments to municipal and 
private schools receiving the voucher payments. 

Municipal and private-subsidized schools compete directly for SNED awards. The distribution of 
awards between these two sub-sectors did not differ greatly. The portion of private-subsidized 
schools receiving awards was slightly higher than in the case of the municipal schools. This rela- 
tionship varied between regions, with the municipal sector receiving a slightly higher share of the 
awards in two of the thirteen regions of the country. Few private-subsidized schools have been 
established in remote rural or urban slum neighborhoods, so the municipal sector is more heavily 
represented in communities with low socioeconomic status. The stratification system is designed 
to assure that schools compete against peers, but there can still be differences within a region in 
the distribution of awards between municipal and private-subsidized schools. Table 3 below shows 
the percentage of schools in each sector and the percentage of awards going to that sector for the 
1998-99 application of SNED. 



: ' \%'V T3Ble r 3: Distribution of SNED Awards 

, ? V Bctwddn Municipal and Private Subsidized Schools, 1998-99 


Municipal Schools 


Private Subsidized Schools 


% of Schools 


% of Awards 


% of Schools 


% of Awards 


69.1 


67.7 


30.9 


32.3 



Source: Data provided by the Ministerio de Educaddn , Santiago, Chile 
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Viewed in terms of enrollment shares, municipal schools accounting for 21.5 percent of total mu- 
nicipal enrollment were winners while private-subsidized schools with 29.7 percent of total enroll- 
ment in this category won awards. 



TDs® OirafODTOiaffloirD Has® ffiw’ SBKI !® 10 

The most important source of information on educational outcomes is the SIMCE tests. The tests 
of Spanish and mathematics administered at 4th, 8th and 10th grade levels produce school aver- 
age scores that account for 65 percent of the total value of the index (including both the cross sec- 
tion and longitudinal comparisons.) A special survey carried out for SNED in 1997 provides data 
used for the indicators that measure the "Initiative," “Equality of Opportunity” and "Participation" 
factors. School data, which do not change greatly from year to year, come from regularly-reported 
statistics and are used for the "Working Conditions" factor, and the rates of student survival and 
promotion used for the "Equality of Opportunity" factor. Other information on within-school activi- 
ties and practices for the "Equality" factor come from the SNED survey. For the final factor, "Par- 
ticipation and Integration of Teachers, Parents and Guardians," information comes from a form 
("Ficha") completed by school directors in connection with the SIMCE tests and from a special 
questionnaire completed by the parents. 

Data for stratification and grouping of schools into homogenous groups come from the ministry sta- 
tistics unit, the "Index of Vulnerability" compiled for JUNAEB, and the SIMCE ficha that is com- 
pleted by school directors. This last provides information on family expenditures on education, 
parents' education levels, and the distance the child has to travel to reach the school. Chile offers 
an unusually rich body of data, especially the outcome data from the SIMCE tests, that is not 
equaled in most other Latin American countries. This has lowered the incremental cost of the in- 
formation needed to establish the SNED system. 



10 Information on data sources for the index comes from Ministry of Education, Hey! and Guzman (1998), op. 
cit , Cuadro No. 1, p. 8. Data sources for stratification are given in Ministry of Education (1998), Reconod- 
miento del Compromiso Docente . op. cit. pp. 6-7. 
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The design and management of the SNED system has been well-informed, sophisticated and 
careful. The Ministry of Education commissioned a body of evaluation studies of both the design of 
the SNED system and its implementation. Studies of the adequacy of the design and the imple- 
mentation of SNED were carried out following the 1996-97 application and another evaluation fol- 
lowed thel 998-99 round of SNED. The ministry commissioned the Centro de Economla Aplicada 
(Center for Applied Economics or CEA) of the Industrial Engineering faculty of the University of 
Chile to carry out much of the evaluation work. 11 The studies aimed to improve the SNED system 
by examining the design and methodology of the system, the implementation process, the infor- 
mation used, the attitudes of actors toward the process, behavioral-motivational factors, the time 
dimension, and costs. 

Following the first (1996-97) round of SNED, CEA was commissioned to conduct two studies 
dealing with not only system design but also the perceptions of the actors involved in SNED 
(school directors, teachers, regional staff) of how the system functions in practice. In addition to 
information deriving from SNED itself, this study drew upon information from in-depth interviews, 
focus groups and surveys developed for the evaluation. Information gathering was carried out in a 
sample of three of Chile's 13 regions, including the large metropolitan region around Santiago. 
Within these, the study stratified the schools for in-depth study so as to include both municipal and 
private-subsidized schools and those that won awards as well as those that did not. 

The evaluation of the design of SNED included a review of international literature on providing in- 
centives for school improvement In this review the evaluators identified the problems associated 
with merit payments to individual teachers; (1) the difficulty and transaction costs associated with 
measuring the contribution of individual teachers, (2) the possibility of opportunistic behavior on the 
part of teachers; and (3) the need to include other variables in addition to pure student achieve- 
ment. On the basis of their analysis of the findings of studies such as Mumane and Cohen 



11 This section summarizes the findings of the evaluation, as presented by Mizala and Romaguera (1999) op. 
cit. The Centro de Economia Aplicada (CEA) is part of the Industrial Engineering Department of the Faculty of 
Physical Sciences and Mathematics of the University of Chile. The principal investigators for the study were 
Alejandra Mizala and Pilar Romaguera, both of CEA. 
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(1986), 12 Hanushek (1986) 13 and Richards and Ming Sheu (1992), 14 it was found that the SNED 
system avoids the main problems associated with merit pay to individuals and incorporates fea- 
tures that international research finds positive. 

Key Findings of the 199637 Evaluation of SNED 
Design and Methodology 

1 . The evaluation praised the establishment of homogenous groups so that schools compete 
with others in similar circumstances. An important recommendation was that the homogenous 
groups be established within each region, not for the country as a whole. 

2. There should be changes in the weighting of factors used in formulating the SNED index. (Ta- 
ble 2 above shows the changes that were made.) 

3. There should be improvements in the factors, especially incorporation of more and better indi- 
cators. 

4. All indicators should be normalized to improve the calculation of the index. 

5. Information-gathering instruments should be refined so as to improve gathering data for the in- 
dex. 

Implementation 

Evaluation of how well the system is being implemented was based in part on questionnaires, in- 
depth interviews and focus groups. Interviewees and focus group participants came from both 
municipal and private-subsidized schools. The evaluators also consulted representatives of the 
Regional Secretariats of Education (SEREMIs) and Provincial Departments of Education (Depar- 
tamentos Provinciales de Educacibn or DEPROVs) as well as knowledgeable people in the central 
Ministry of Education. The findings and recommendations of this phase of the study were the fol- 
lowing: 

1. There is notable consensus that a system for evaluating school performance is a good idea, 
although there were suggestions for improvement. 

2. Respondents approved linking the evaluation of performance with the payment of monetary 
awards. 



12 Mumane, Richard and David Cohen (1986). "Merit Pay and the Evaluation Problem: Why Most Merit Pay 
Plans Fail and a Few Survive". Harvard Education Review , vol. 56, No. 1 , pp. 1-17. 

13 Hanushek, E. A. (1986). "The Economics of Schooling: Production and Efficiency in Public Schools." Jour- 
nal of Economic Literature , vol. 24, pp. 1141-77. 

14 Richards, C. and S Ming Sheu (1992). "The South Carolina School Incentive Award Program: A Policy 
Analysis." Economics of Education Review , vol. 11, no. 1. 
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3. Stratification into homogenous groups so that schools are competing against equals is highly 
desirable. 

4. Respondents recommended modifying the weighting of factors in the index and the way some 
factors are measured. 

5. Respondents urged that the evaluation system be transparent and that schools have access to 
the information used to determine which schools receive awards. 

6. Regional and provincial authorities recommended that they have greater participation in the 
process, especially in identifying the homogenous groups in their regions. 

7. Schools that won awards suggested that, in addition to the money awards themselves, that 
there should be greater publicity attached to winning the awards so that parents and the com- 
munity at large should be aware that their school was a winner. 

Changes as a Result of the Evaluation 

The most important changes between the first two applications of SNED were: (1) the shift in 
weightings shown in Table 2 above, which reduced the weight of factors based solely on SIMCE 
scores from 70 to 65 percent and increased the weight of the "Equality of Opportunity" factor and 
(2) formation of homogenous groups at the regional rather than the national level. In accordance 
with the evaluators’ suggestions, the SNED system made technical changes to standardize all val- 
ues used in calculating the index and to use more and different indicators for the “Initiative” and 
“Equality of Opportunity” factors. Certain other changes affected only the basis for scores for sin- 
gle-teacher and multigrade rural schools, free-standing pre-schools, special education schools, 
adult education and prison schools. 
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School Directors’ and Teachers’ Opinions of SNED 



The evaluation study gathered qualitative information through interviews with school directors and 
used survey questionnaires and focus groups to obtain data from teachers and school directors. 15 i , 
The purpose of gathering this qualitative information was to assess: (1) the perceptions school di- 
rectors and teachers have of the SNED system, (2) whether SNED contributes to improved teach- 
ing performance, and (3) the degree to which SNED encourages competition between schools to 
improve educational quality. The findings provide interesting insights into reactions to SNED at the 
school level, both in schools that won awards and in those that did not. 

Views of 8cfo©©3 ©omectas 

School directors’ perceptions are interesting because they are both teachers and, in their supervi- 
sory roles, informed observers of how teachers react to the introduction of merit awards. The inter- 
view-based information provides insights not available from the survey. 

The general attitude of school directors toward the SNED system is positive. This includes direc- 
tors of both schools that won awards and those that did not. The directors, whose point of view is 
somewhat aligned with “management”, see the SNED evaluation system as a way to recognize 
differences in the performance of schools. The design of the system is generally satisfactory, in 
their view, although they are somewhat concerned about the heavy emphasis accorded to SIMCE 
scores through the combined weight of the “Effectiveness” and “Improvement” factors in the SNED 
index. In this context some directors expressed the view that some schools practice selection of 
students (even though this is against regulations and the SNED system itself penalizes such be- 
havior.) 

Directors are in favor of a system of monetary rewards for teachers. In their view teachers’ salaries 
are too low and awards under the SNED system represent a partial step toward rectifying this 
problem. Virtually all directors approve of making the awards to school establishments rather than 



15 The information on directors’ and teachers’ views of the SNED come from Mizala and Romaguera (1999), 
op cjt .. Sections IV and V, pp. 18-39. 
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to individual teachers, so as not to undermine teamwork among teachers in a school. It is inter- 
esting that, although SNED regulations reserve ten percent of a school's award to be allocated to 
"outstanding” teachers at the director's discretion, a substantial majority of directors said that they 
distribute this ten percent on the same basis as the main portion of the award-equally across all 
teachers at the school. This tends to support findings of earlier studies of merit pay to individual 
teachers that the difficulties of making and justifying individual awards and the negative effect on 
teamwork and feelings of solidarity in the school outweigh any positive effects of individual re- 
wards. The directors indicated they would oppose a system of rewards to individual teachers. 

The prestige of their school and its ability to attract students is important to school directors. Some 
directors of winning schools made efforts to publicize the awards but others did not. There was 
skepticism about whether the SNED awards would have an impact on parents' decisions about 
where to enroll their children. More important, in the directors’ opinions, was a school’s long-term 
reputation in the community, especially its role in developing strong values in the students. Direc- 
tors feel that families judge a school on the basis of its values, the commitment of teachers, what 
other families are choosing, and pragmatic factors such as distance from home to school, as well 
as achievement scores. The SNED awards cannot alter more deeply-rooted reality, in their opin- 
ion. 

Directors’ Comments on Teachers 9 Reactions 

When asked their opinions about teachers’ views of SNED, the directors gave a somewhat differ- 
ent picture. In the directors’ opinion, teachers have more reservations about a system based 
heavily on standardized achievement tests. They mentioned a “teachers’ culture” (cultura docente) 
in which teachers see themselves as motivated by values and commitment to their vocation. Ac- 
cording to the school directors, teachers see themselves as doing the best they can, even under 
difficult circumstances. In this view the SNED awards are seen as acknowledgement of what 
teachers are doing in any event, rather than as recognition of excellence or special efforts. To a 
considerable extent directors’ comments indicate that they share these views. 

In the same vein, directors emphasize that teachers would object to evaluations of their work 
based solely on the SIMCE tests, which do not take into account important factors such as student 
learning in areas other than Spanish and mathematics, nor of their role in inculcating values and 
attitudes in the students. This criticism has been voiced by the teachers’ union, although the union 
has not opposed the SNED system. 
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Some directors doubted that SNED would motivate teachers to greater effort. The evaluators 
quoted statements indicating that the contribution of SNED to teacher motivation is “fragile" be- 
cause of the strong teachers' culture or cultura docente. Some directors felt that what would make 
a difference in teacher performance would be a general salary increase of significant magnitude, 
greater public recognition of teachers’ efforts, and substantive participation of the teachers' union in 
design and implementation of education policies. In the directors' opinions, teachers tend to be 
convinced they are doing a good job on the basis of their own professional commitment and val- 
ues, regardless of monetary awards. This is borne out by responses to the teachers' questionnaire 
discussed below. 

Survey Data on Teachers’ Reactions 

The teachers’ survey found that teachers tend to accept “the Chilean model of education”, includ- 
ing the voucher system and co-existence of municipal and private-subsidized schools. There is not 
much difference between the attitudes of teachers in award-winning schools and others, but teach- 
ers in private-subsidized schools showed significantly greater acceptance of the “Chilean model" 
than those in the municipal schools. 

There was fairly high acceptance of the concept of evaluating schools and awarding prizes on the 
basis of performance, especially on the part of teachers in private-subsidized schools. Teachers in 
both municipal and private-subsidized schools that won awards agreed with the statement that 
such awards can bring about improvements in education. Teachers in losing schools were less in 
agreement with such statements. 

Teachers' responses indicated a generally positive attitude toward the design of the SNED system, 
although a significant percentage found the design complex. There was some concern about the 
indicators used in the SNED index, including both the categories and their weightings. In general 
the concern focused on the relatively heavy weight given to SIMCE achievement test scores. The 
concern was greater on the part of teachers in private-subsidized schools that did not win awards. 
Some teachers would have preferred more use of qualitative indicators, although at the same time 
they approved of the relative transparency of the present design. 

The survey found that teachers evaluate their own work quite highly, regardless of whether their 
school won an award or not. Seventy-four percent of teachers in schools that did not receive 
awards classified their own teaching performance as good or very good. This finding coincides 
with comments of directors that teachers are fairly complacent about the quality of their own work. 
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The survey was administered in 1997, shortly after the first round of SNED evaluation and awards. 
It found that teachers did not have very complete or accurate information about SNED, especially 
teachers in schools that had not won awards. This, together with similar findings about directors’ 
knowledge of SNED, led the evaluators to recommend strongly that the Ministry of Education un- 
dertake an information campaign to increase understanding of SNED. Unless there is full and ac- 
curate information about the system, it cannot have the desired impact on the quality of teaching. 
As a result, the ministry has taken effective steps to implement an information campaign. 
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Conclusions and Implications for Policy 



The SNED system of merit awards to schools has been carefully designed and well implemented. 
As SNED enters its third two-year round of measurement and awards (2000-2001), it appears to 
be accepted by key actors in the education sector. The managers of the system in Chile's Ministry 
of Education have been concerned to evaluate how it is progressing and make adjustments as 
needed. It is too soon to say whether this approach to providing money incentives to improve 
teaching performance is having its desired effect in terms of improving student learning but, with a 
cost equal to roughly one percent of the vouchers paid to schools and indications of positive impact 
on the attitudes of teachers and school directors, the experiment appears to be worth the effort. 

As a system of awards to establishments rather than to individuals, the SNED system avoids some 
of the problems associated with merit pay for teachers identified by Mumane and Cohen in their 
important 1986 article. 16 That study found that transaction costs associated with measuring the 
performance of individual teachers are an important reason merit pay schemes do not work. Since 
SNED takes the school rather than the teacher as the unit of observation and award, it avoids the 
difficult measurement problems arising from what Mumane and Cohen call “the nature of teachers’ 
work" and the need to provide objective indicators. 

There are three broad categories of incentive systems to improve educational performance: (1) 
systems of rewards for good performance, which can include “merit pay” to individuals for excel- 
lence in teaching or merit awards to school establishments; (2) systems offering “choice” that pro- 
vide an alternative to public education and introduce competition; and (3) systems of external stan- 
dards that provide a yardstick for measuring how well schools are performing and pose varying 
degrees of “threat” to those that are performing poorly. Chile is applying all three of these: the 
SNED system of merit awards to schools, its voucher or subvention system, and the SIMCE test- 
ing system that provides public information on how well individual schools are performing. It is well 
to bear this in mind because it will Complicate any eventual attempt to attribute improvements in 
education to any one of these interventions. 

Since the SIMCE tests were already in existence, the incremental costs of creating the information 
needed for SNED was very low. They included only the cost of analyzing the data for the SNED 
index, plus the cost of the survey used to gather data for some of the non-SlMCE variables that are 
the basis for other factors in the index. This means that a country that has no system of testing that 



16 Mumane, Richard and David Cohen (1986). Op. cit. 




« 28 




implementing School-based Merit Awards; Chile’s Experience 



measures all schools (instead of a sample) would face both the costs developing the testing sys- 
tem and all the negotiation costs involved in establishing it and developing its credibility. 

The managers of the SNED system have taken pains to evaluate the system and make changes 
to improve it. Following the first round of SNED, there were two evaluation studies: of design and 
methodology and of different actors' perceptions of the system. Following the second round of 
SNED (1998-99), an evaluation of perceptions was again carried out, which made it possible to as- 
sess changes in understanding of the system and perceptions of it. After the third application of 
SNED in 2000-2001 the Ministry of Education plans another study that will evaluate whether there 
have been changes in educational processes as a result of SNED. It is still too soon to evaluate 
whether SNED has had an impact on student outcomes. 

There has been criticism of the low financial value of the SNED awards. Low awards tend to avoid 
problems of opportunistic behavior on the part of teachers, such as focusing their efforts on the 
subjects (and possibly on the kinds of students) that would lead to the greatest increases in the 
performance indicators. Some beneficiaries argue that the awards should be higher. This, how- 
ever, “raise the stakes” associated with SIMCE and other factors in the SNED index and possibly 
bring forth opportunistic behavior. The evaluators conclude that it is probably desirable to start with 
low-level awards, at least until the system is refined and well established. 

Reserving ten percent of the awards to reward teachers who have made the greatest contributions 
to school performance was intended to encourage all teachers to make their best efforts and 
thereby avoid free-riding on the part of some teachers. A majority of directors, however, have de- 
cided to forego this option and distribute the full amount of the award on the same equitable basis 
as the other 90 percent. This suggests that the transaction costs associated with determining 
which teachers should receive extra payments outweighed, in the directors' opinion, any benefits in 
terms of avoiding free-riding. 

Information gathered from both school directors and teachers emphasized the importance of feel- 
ings of teamwork on the part of teachers, especially in schools that won awards. 17 Directors 
praised the system of awards to whole schools rather than to individuals because this avoided fric- 
tion within the school team. In schools that won awards, the reaction among teachers was that it 
was a recognition of the good work of the team. Although the subject goes far beyond the scope of 
this paper, these observations suggest that there is a kind of "social capital" that develops in suc- 
cessful schools. 



17 Mizala and Romaguera (1999), p. 24. 
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A disturbing finding of the evaluation was that most teachers had very little knowledge about 
SNED. If merit awards are designed to elicit improved performance on the part of teachers, it is 
essential that the teachers be fully informed about the system. The publication Reconocimiento al 
Compromiso Docente (1998) represents one step to provide better information about SNED. The 
ministry is preparing an extensive program of information and training for regional officials in ad- 
vance of the next iteration of SNED in the period 2000-2001 . 

A problem that is not addressed in the evaluation or elsewhere is how the SNED system will func- 
tion on a dynamic basis in the future. The main evaluation of SNED was carried out following the 
1996-97 application of SNED (when levels of information about SNED were found to be low.) It is 
not clear what the experience will be following repeated rounds of measurement and awards. In 
the school directors' opinion, teachers in schools that won awards in one round of SNED but did 
not win in subsequent rounds would feel frustrated. Once a school has achieved high levels on the 
SIMCE tests, it becomes more difficult to score well on the “improvement" portion of the index. 
And if schools make efforts to improve their performance but still do not win awards, what effect will 
that have over time? 

A related question has to do with the appearance of opportunistic behavior in the future. As SNED 
becomes better known, teachers and schools directors may devote increasing efforts to improving 
their schools’ scores in ways that do not truly benefit students. These might create distortions in 
teachers’ allocation of time among subjects, encourage schools' efforts to screen students, or mo- 
tivate schools to urge poor students to stay away on the days tests are administered (as has been 
observed in countries where “high-stakes” tests are used). 

With regard to both these points, it is important to note that the SNED system considers not only 
SIMCE scores and changes therein but also other factors that make up 35 percent of the index. 
The inclusion of other factors in the index provides ways other than scoring high on SIMCE for 
schools to win awards and will reduce propensities to engage in opportunistic behavior with regard 
to the tests. It is also the case that the variables included in the factors comprising the index (and 
even the weights of factors) will be the subject of continued evaluation and possible change in the 
future. 

Chile’s merit award system is compatible with the teacher compensation strategy recommended by 
Odden and Kelley (1996). 18 Some observers, including school directors in Chile, argue that what is 
needed to stimulate better teaching performance is a general salary increase for all teachers. But 



18 Odden, Allan and Carolyn Kelley, Paying Teachers for What They Know and Do. Thousand Oaks, Califor- 
nia: Corwin Press, 1998. See especially pp. 1 12 ff. 
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Chile’s SNED, by providing salary complements based on "high standards, high performance," as 
advocated by Odden and Kelley, is currently the best developing country example of recom- 
mended international practice. 
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